To access the contents, click the chapter and section titles.
Oracle Performance Tuning and Optimization
(Publisher: Macmillan Computer Publishing)
Author(s): Edward Whalen
ISBN: 067230886x
Publication Date: 04/01/96
Chapter 5 Benchmarking
A benchmark is a test used by computer hardware, software, and application vendors to test the performance of a system. Benchmarks are designed so that the same test can be executed on a variety of hardware and software platforms. Because the same tests can be run on a variety of systems, the results can be compared to give a relative indication of performance.
Because benchmarks usually test a specific function, they can give only an indication of the systems performance in that area. You may find a system that performs extremely well on an OLTP benchmark but does not perform well in a decision support environment.
There are essentially two categories of benchmarks: standard and custom. The industry standard benchmark is typically run by the computer hardware or software manufacturer; the custom benchmark is usually used by the end user or integrator.
The industry standard benchmark allows computer hardware and software vendors to compare the performance of their systems against other vendors systems. These benchmarks have been designed by the industry as a whole to provide a fair and level playing field. This chapter looks specifically at the TPC (Transaction Processing Performance Council; in its early days, the council shortened its acronym from TPPC to TPC), which is dedicated solely to database benchmarks.
The custom benchmark is used extensively throughout the industry; it is designed by the user to compare systems with a test that closely mirrors the systems environments. Custom benchmarks are also widely used in the computer industry to allow the manufacturers to test specific components and make improvements or overcome deficiencies.
Custom benchmarks are also used by publications to test certain configurations of hardware or software. These custom benchmarks usually have strict configuration requirements such as hardware limitations, OS tuning constraints, and RRDBMS layout so that they can provide a comparison of a specific subcomponent of the system.
Introduction to Benchmarking
A benchmark is designed to test a functionality of a system and to report its performance. Some benchmarks are designed with general functionality in mind; others are quite specific. Database benchmarks are typically divided by transaction types. By far, the majority of database benchmarks are OnLine Transaction Processing (OLTP) with a few decision support benchmarks now available. Of course, custom benchmarks span the entire range of database transaction types.
This chapter first looks at the industry standard benchmarks available from the TPC. Then it looks at how you can design your own benchmark. The industry standard benchmarks are useful for giving you a rough estimate of how a particular system compares with other systems on the market. This comparison may allow you to shorten the list of systems you need to test before making your final decision. Custom benchmarks are great for giving you a real feeling about how a particular system will perform in your specific environment.
Industry Standard Benchmarks
Industry standard benchmarks are designed and maintained by organizations made up of computer hardware, operating system, and database-related companies that have a stake in seeing that the tests are fair. In the case of the TPC, these companies also police each other to make sure that fairness is maintained.
Because the organizations are made up of so many competing companies, the test results are usually seen as fair and comparable. Once the benchmark designs are completed and companies begin to run the benchmarks, competition sometimes becomes quite fierce. Each company wants to be seen as publishing the fastest and most price-competitive benchmark.
A phenomenon that is quite apparent in the database performance arena is that systems previously thought of as low-end are increasing in performance at an incredible rate. PC servers, for example, are achieving database performance that had been reserved for mainframes just a few short years ago.
The following sections take a look at the TPC and some of its benchmarks and give some information about where you can get the results of these benchmarks and how to interpret them.
The Transaction Processing Performance Council (TPC)
The Transaction Processing Performance Council (TPC) was founded in 1988 by eight computer hardware and software companies. At that time, there was no industry standard benchmark on the market; companies had a difficult time trying to compare themselves to other companies.
The TPC was founded with the goal of defining transaction processing benchmarks that provided objective and verifiable performance data. The objectives defined by the original eight companies are still followed today. These objectives are to provide industry standard benchmarks with the following qualities:
- Complete and thorough specifications that allow anyone to implement the benchmarks regardless of system architecture.
- Objective and verifiable results that do not handicap any particular architecture.
- Full disclosure of the system configuration, cost, and benchmarking methodology. This allows results to be independently reproduced.
- A fair means of comparing one system against another.
|